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| speed of terminal (efther. 10, 15. or 30 characters. per’... fru: 
“+ seeond)’s - -2) pe. of application (for each individgal ~ 7 
. service piques ).fr.The, datarare grouped according-to. .-.  -Aj:. 
.' these criteria’ ajid: cumulative: ‘frequency distributions are: 

a ae - computed for. each of 14 parameters of the model. Non- _ ed 
ail oo (parametric | tests are used to deteriritie the significance ~~ 

a eu ac. he BOE differences An the diatribubtons for Re sets Of. oo. 
= . = = data. pe ee ar So Pa = Se 
6) fe Oe gE ee oe ee see i ee 
ee - The Teendetees. ‘itself ‘is: the major contribution of. . = 
oe _ the study, providing, as it does, ‘a ‘quantitative: way to - 


uy pnvestigate | a variety. of phenomena associated with ib, 
Cele interactive comguting. The most interesting Specific 
finding. from the the data collected is the increase:in « |° 
youtput data length as “the. terminal Speed. increases, ihe a 
a 1% ; ' ; ¢. 9 . . Pe * aa 
tg Key Words: Computer - performance evaluation? poner 
oc. fAetors;. interactive computing; | man- ~frachine . eee 
Rae raeel gents: performance measurement; Sete ee 
te oe This orm addregbes't the nee of Steen we = Vo 
ce computing» i me Surement both of the computér system providing +. 4. 
. ‘the service (including: the ‘communications. syst’em delivering 1h), 3 ee 


. and’ of the: humans using At. Anteractive. apeluee eo elena ae of 
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ee ea, include the users asa System Gomponent. Whose,- —°"" 
_. per ormance affects. the overall system. Traditional computer -.-. 
“system measurement.has° failed to“chnsider ‘the needs: of dndividual 
- Computer system users “(focusing instead on overall performance’ .|.... . 
.. Méasures), nor has: it considered =the impact,of. uSer-performance:: 7 
_ on. computer’ system perfarmance, »* 4 OP gt he eg rc 


‘In this dissertation, a measurement approach is described ioe 
and. appliéd’ that does address both these méasuremén fore &: 


interactive’ computing. A.new imeasurement tool devel ed: atthe °° 
-  —.. National Bureau Qf Standards is employed to measure actual. Ss ee ee 
__ interactive’ sessions,or convérsations on.-a timesharing computer. ae 
' Fcsystem .over a period of three years, 6n1y Successful and-) 
arc “representative conversations. are considered. in this Study; — =. 
. aborted or otherwi'se ‘erroneous. onversations were: discarded, The,’ 
_ basic data gf interest for the "good" conversations are the ~~ 
» number .and rate ‘of. characters sent ‘by user and system,’ and . 
“latencies ari délays prior to and during transmission by.either 
party... These data are'fit..to &@ model: of} user-computer ee 
interaction which distinguishes ‘between Stimuli or input from te. 
user, acknowledgements fron’ the- system: (which only indicate that. — 
| 6S Service request’ has been received) and’responses from the 
ar System (which. contain: meaningful information) . he 
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‘ . Vv 
i | Statistical procedures are developed and ‘employed—6 analyze “ 
+, . , the significance of the results when thes data are ‘fit to.this. ° 
8 a; model of user-computer interactign and then- royped atcording to -" ' 
ar ae Various ‘criteria’ Analysis of the dat& co ‘Sts. of: grouping. - 
“-aevording’ to twq.independent, criteria: 1) maximum operating line 
- 4%. Speed of the terminal (either 10, 15 or 30 characters pen w te 
o 8ecpnd);. and 2)- type of: application,: as indicated’ by the a ee 
2 ‘software-ssubsystemiinvoked by each individual service request ‘in 
_ the conversation. The data are grouped according tethese | -:. 
‘criteria and cumulative frequency distributions -are computed for con 
JN each of 14 parameters of thexmodel, both for'ali observations in’? 4 
y thevgroup,. and for the medians of the seto ally apprie —— 
va (observations in each canversation (as.a way of batching the data 
"+ to eliminate any, evident serial correlation between successive . 
observations ina. conversation) .. Non-parametric tests dre used - 


: bo. to determine the significance of: differences ‘in the. frequency: OS eg the 
- distributions for different sets.of data," .-. Sea 2, —_ 


’. 


..'. In the case of grouping by terminal. speed, significant. 

. ‘ information is obtaihed characterizing. both-yser and system... re oe 
performance. .~ In the ase of grouping. by application’, the: data Cae a 

4». are not analyzed’ soexhaustively;.,. rather, the potential types of ee ce 

“. -analyse's FT eonady ooo cond mot atede fier: spplicabions POT. Ov hy 

Enis measurement approach: ate. also suggested. ’, ~ ae eee a ee a 


“oa . \The «methodology itself ifthe major contribution “of the ee ee 
st a) ores cine does, a. quantitative Way to investigate. a. 
_VvV@riety of phenoména: associated with interactive.computing. ' The- 


most: interesting. spebific finding from the.data collected is the. og 
 . Mneréase in Output data length as the terminal speed ingreases. = =v 
5: * Ths observed increase,,: over. the range f. speeds studied,’ - Le op os 
sip an 
a a 
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. sontrasts. twith: th. sbsenved Sere ‘in a, previous bet ha 
study with a range of speeds higher than’ those in this study, “=... 

7" " "These results lead to the postulation of an.upper limit on. ‘the’ es 

ue ps ubllity of teraiinal. pada to ugers (with surrent modes of. a 
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“Chapter: 2. “provides. Sadkercurd: ‘on “the measurenene: cof. 
ve .', interative computing. The difference between. traditional. 
ire m@gasurement techniques st at focus on overall. system performagee - 
‘and new Gr tnd ivi that focus-on the- service, provided to and the .- _ 
“behavior f individual: users ‘As: Seale oe “Thepimpor tance f the.‘ " 
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URE in inieractivd aystyns ts stressed; followed bya review of che 
_ the irelevant system char cteristics of. users considered as systep 
", , components , Several ‘models’ of. user-computer, interaction -are " 

“presented, including -a fecently, developed one. tMat ig. used in the. 
"analysis ‘of@the data collected in the Study. ° The ch@Bter. ~ * -» che 
concludes with a consideration of the data-colle@tion.’. eee 

| mequirements.of the model. a co a re er : 
ee ia ae 2 “ ao a e 


gE “8S Chapter 3 deals with. the: actual data ao ection occas a a 
ae for -this investigation. A riovel data colin instrument, the 26 
oot Natdonal Bureau of Standards’ Netwonk Meas ement. Machine, ig -' 
ie descr iped 'aldéng. with the dat analysis software. available for use’ 
with it.) This ‘instrument. ; ‘Used to measureginteractivel use-of.. 


thre NBS. Univac’ “1408 -ov'er a ree ygar period. | Only! "normal": Boas wee 
recorded ¢tonversations are’ nalyzed, 


. ? \ 3 +. The folyme of data t sy 
. . . ° collected ,and the screening criteria are: d: Setibed, 7. 


. @ , Mee’ 3 ot ieee ae a 
Be " ‘Chapter 4: deals. with the analysis procedurés for the data So) 
“. . after it has beef collected’, Two different. ways. of Grouping the. 

ct. data aré describeds ~ (1) according to the speed.-of the inter# - | 
oe taetive terminal, used (110 bits ‘per second, 150 ‘bps or 300. bps),... | - 


os) ahd (2) aegording to the soft me subsystém invoked by the user Se 
>’. | at. each- stage tn ‘the interagfive session, The issue. of data _ 
'( independence betweeh-.suece Ging observations in the same. _ ae 

nuversation (sergat “oF augoe correlation) is addressed and test t 

résults are” presented.) THe “skewed nature of the data makes it: ge te 
' impossible to: apply statistical tests based on the nermal - mo, 
distribution. . Two non-parametric. teSts;. the Mann-Whitney U-test — 

7) and the: Kolmogorov-Smirnov test, appear to be suitable to analyze . © 
ol: data ofethis type. The bases of these tests arg described.» oy ‘ 


phe 


“ JF : ? : : . ota ? me : . ef sy ote 
os In Chapter 5 the results of analyzing ‘thé data.according-to .. - 
the selécted ‘madel of ‘user~computer .interaction, -grouped - . 

*  ge@cording to terminal. speed, are presented.’. Median and 
mo’ “3 90-percentile statistics are tabulated for each of fo 
"=: parameters for the"differeht speed termingla’ mia ty 
". *,: diseusselW. |. The differences between the paratteter 
|. ‘for the AJO bps- and 300.-bps data are tested for 

J using the Mann-Whitney. U-test. 


Biss t\and. the -Kolmogor 

Peas. “ie, FesuL.us. presented . at ee et i. : 

a ae : . a ‘ aa ~ Bae 7 7 a me os ~ ce . ‘ . oe 7 o as 

ae cae . jny@hapter 6. the results of analyzing the data according to 7 
woe ee £h neon 


1@ Same“model', but ‘grouped according to software system invoked ~ -:: | 
Cie * by*.the ager, are presented., Median and 90-percentile Statistics > = 
oo ae tabulated for each of.thg fourteen. parameters f@r the™: 


ots odifferent. software systems, ‘and the results are disdussed; 


ps) ae os ‘Chapter 7 concludes the Study with a summary of methodo- eo, 
_: 7 « logical contributions and empirical findings, discussion, of. ayfas || 4 
“pd Of application, identification of 1igitations and suggestions for.":.. , 
| sfuture work in.the’ area of. interagtive computing measurement, ° + 
Primary areas of possible application include benchmarking in: 
a System procurement, ‘gathgring user- parameter’ for use if = 
“Specific system design, and tuning interactive Systems. dn fact, 
<b e approach. has already been applied in procurement Situations. 
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Sy 2, o- LMEASUREHENT ‘OF "INTERACTIVE, COMPUTING . 
ue . In revert. years, incheasing nanbers ‘of. people. have eau té 


2B J ode, Rather thah submitting jobs in a ‘batch mode and waitin 
Tpke We tee days) f : 
ee oe ag, OF tinuous y. on a transact ion-oriented basis. This trend gives: 


; the ‘Predominant Fopmvaf-gonpuber access - in tire, future, ue 
Ho. Ae 
oe ‘The* history: of “int? 
mgr inability to. come to grip 
‘opt dnter- -~rélationships 9 vb hy 
‘, constitute such systelt 


ywe ‘many: diverse’ components. that 
AS: interactivé.systems have become : 


ces “and ‘moré notable, Ge, is + resérvation systems. that simply’can not: 

hae’ Handle the Toad)... "This, there is a need for: quantitative data 
© “* describ ng the - veils heen ene of the. various. components of 
interactive’ com ut Een s and. relating. the performance of . -: 
Meee component si.’ 
ata is needed by ygten’ et ee ‘and implementers. for use in 

'. “getting. design goals, as wel 

s al use in. ‘specifying requirements» 
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In this chapter, we gonstder the rrr goals” of. 
traditional System measurement: and evaluation as :compared ‘to tiie. 
measurement and“evaluation of ‘interactive Systems: We see that’: 
_»  , the focus is, of necessity, q ite different. System managérs 


. 
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. efficienty,; while individual users are primarily concerned pate 
*-. the -level of servide they each. receive, Frequently, tradeoffs” 
., must be made between efficiency. and level of*service,to 
_ «©, dindividualvusers. Also, the performance of ‘the. users themselves. 


; ee a need to. consider the user as a system. component for which 
oe me "operating data" Ps required ‘Hence, we review the. relevant: 
SUEY Sadie acer of. human operators.” ‘We then describe several 
- mod@ls that can be: employed to characterize the interactive 
a7 situation, including-.oné that - will ‘be used for.the analysis. of 
sa. the data: collected in’ this study;. We conelude with an jntro- 
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ease computer. enn te eo ee ° 


won a. et a: a ee ce - 


ow "Measures. of. Service , 


ae — Mubh: shecntien has been d regted : in iP ebeon ts eoane’ ates 
e ..- ‘in€asurement and’ related quant tative. evaluation: teshniques ‘asoan’: 
2 - important part ‘Off : the selectfon and. iinprovement of the hardware 
s "and software of computér systems. Both hardware and softwane ~. 
wes. “tuning: of the system can be undertaken in responge. to the 
‘information gathered, The. overall sesults of such ea ney ‘be: 
ee ie increases. in system throughput. ° 
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use © computers | ‘through interactive terminals inta conversation 1 


impacts tha functioning ofthe computer system, Thus, there as. 


dh. the results, ‘users interact. with the com Uteh 


. 7, every evidence ofcontinuing, and -it appears that this. will, be | " 


mactivé. systems has. ‘been marked by: ‘an “6 ea 
Hwith the design. problems raised. by the 


an E “Y , 
en clargér. and more ambitious , 801; do, have the failures. become larger. 
othe ‘pérformance of the. overall system, This... 2 


as by- users: 2Ang ven ‘procurens yo 7 


Seek to optimize the’ ete peration of the system for maximum 


a duction to the measurement. tectiniques - suitable: for inve. aaa . 


how, oe . ow _ €. i wa a . a 


° : “3 ea : . g : ‘ 2 : ie te ‘ ,, 
= ms - oe Bee ‘ Pa 7 aot ¥ oe - : y oe. , ee ee : - | ‘ : - * P é Pet zi re oy 
Pe ee Pee ok *. - ek Se OS oo esa “se Se a aaa . in 
pot ou. The basis forme Suring success in performance evaluation: ©: ... cat 
he's. Of this type fe sone heasure ofthe costeerfostiveness ee then. 


-. System before and‘ after the changes. The cost-e¥fectivenéss ©... ‘ 
‘Calculations usually are based. on thetime td run some given set...” 2 

4. Sf Jobs. (a benchmark"): and the ost ‘(purchase ar-monthty rental) . - oes 

» th) of the \partioular configuration, Thus, cost-effectiveness ik | a2, pe, ai? 2 

oh really asratio’ of ‘throughput to cost, and. the measurement. angplyst” ao @ 

oo ' Ls presumed ‘to be indifferant’to the choloe between a.rediokfon. be Be 
“ In cost. for identical. throughput, or ‘increased throughput for > ~ 

- identical cost. The t me“to Process the. Benchmark ‘is important 4 
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ie ec ca ‘ we Veg wo: se he. % a Fe eg getty 4 : : os 
a leotee eduah users, howeVer, May*got be indifferént to. this. i 
 tradeoff*of cost versus time. | For them,.service is the: keys 08 
. factor / not throughput. . In -cohtrast to the evaluation of. © 7+ Ce 
J heffiet ney, whicl’ is.concerned with the time and cost-to run ‘a, | 
. grovP of Jobs, service evalugtion-is concerned with the time and — 
7 -@ogt {0 shungeach individual. job. ‘Actually, the times of congern .°.. 

3 are of different types,. | Efficiency iss concerned: only with, — ie cs 
running or, execution times, while Service evaluation.is concerned’ | es A 
with total elapsed time.** (In most multi-programming ‘systems; --the — 

ie. elapsed tine. fora job is considerably, greater. than the run,. .. 

» 8 hime). Thus, theevaluatign of efficiency 1g based on..the 7 es ae 

“y Measurement of the internal funetioping of a Computer system asa... °° 
_ .- whole, rather than of its external: manifestabions, taken ce 
tala -individually®  , fog ae Te SP Dee Gg gk eg ae ee 
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ie The Goals of improved efficienoy and: improved. service may 7 
well be. at odds-with one another, “An improvement. in internal. . a 
. performance: does not. necessarily imply. an improvement “in. ervice!: ee. 
Indeed, the Opposite may be true. Frequently, itis possible to 
“improve Service only at the ‘expense of efficiency. ewe LR 


ae eons service, ‘the methodology of cost-benefit re 
- analysis is“ffore appropriate. than that of cost~effectiveness * 
» (Cotton, 1974; ‘Streeter, 1972). Cost~-henefit analysis... °°. 
“recognizes ‘that Jobs may have: a time-value,.and.that a- simple. - 
' de + In asense, —¢. . 
Measures of service also must. be concerned with the Standard ~ rn 
| deviation‘ ofgrun. or response time.. When. measuring performance, . *.°°. 
1 jobs run Has a than “average Cancel out+the effect of more . 
te 


me te. rypes. of Jobs,~ geceptable’ and unacceptablé. .No further me 
-. 18 a§eribed>to:jobs run more rapidly than what “is deemed’. a 
"aeceptable": ' there -is no averaging between ‘jobs ruh: - 


| The.-‘mast common performance measure applied to ‘conver=".\. ae eae 
me sational Isystems.is "respons: time.", This ‘measure is most ee 2 ey 
«. o. frequent By defined. to be the ‘elapsed time from the end-of the | . | ae 
,- user's }nput to the -beginning of the system's regponse (National js gb? BS 
cae ee a eee ee fe et hee 8 hh 
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* Other measures of. the system's respohsiveness have been: \. 


a Proposed ( Atrdms,. Lind ‘and' Pyke’, 1973). . 
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“ae Theugenenar oat of. most system désigners has. ‘been 


“vhowever, gystems' ave been designed’ withatit a, ‘particular’, 
time- goal ‘in arepe 


_ gehe domain: of. ‘enginéering-oriented psychologists, some & whom: 
have begun | O.: address. an problem in, recent ‘years. :Tn. the. 
following’ Sedation, ‘we revi 


ry 


ew the functional | oharacteristics of 
users - wee are. relePant to the puri of. cpeerag hive: systems," 
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ae User: ‘Characteristics. ee = as a oo ee ae 


Considering the a lea of: the user tin the design: and: 

_ performance of interact 

7ts known ‘about user characteristics. . The traditional appraach of. 
‘the human factors engineer is to regard the us@r as another ... 

ea component ‘and*to seek to optimize the tOtal . system in 

--consifierations of the capabilities ‘and Limitations of thé human - 
“gomponent (Van Cott, 1972).. Too often, though, interactive. 
systems ‘have been assembled by hardware and software specialists. 

without’ any. asSistance from euch human factors eneineei es « 


Bee. « “The, role of. the yser Ay ine Siaevints egubane 1s ‘both as ‘a . 
“source and ‘Sink’ for information. As an information. Sours | ‘the 


“user is.characterized by .the- rate at, which informatt{o Is. entered. , 


. (including both mean and burst characteristics), Dy ene latency 

of new inputs following'receipt. of a- computer « response, and by 

_ the nature of the information entered (e.g.,. the software system. 
-used). ‘The role of the user as.an- information. Sink is only  § °- 
measured indirectly. in terms of successive latencies: and. ee 
in source information: content. User characteri$tics asa sink 


—" » are such as to place demands on,the system. in terms of its 


/ hesponsiveness and the. vename a ede Ay 
; - 


Serert Input Rate a. ae | a 


User gtyping: dupabihities have ‘been Dees, extensively, as aa 


part. of the .nhuman factors of data entry devices, and procedures , 


_ (Seibel, 1972). For high-volume entry of redun@ant data such as. ~ 


. English text on typewriter-~ like keyboards, spe test rates of’ 60- 
o--words' (300 characters) per minute are.quite> cal 

* limit of about 100 w.p.m. in: production situati 

..60 to. 0150. W,p.m.° corresponds to’ ® range of 5.to- 12.5 keystrokes 
per ‘second, «(These are sustained rates. which can: be exceeded by 
. "burst" rates"for,brief interval4). ‘Uhskilled bias evita 
ee texk ave abolt “|. stroke Per second, \: 


‘ : ss : - x a 


“For Ke ypunehiee? al Gate of 170° Chic esters per minute’ (2, 5 
‘ehsracters per second) is reported .as a good estimate of the mean 
rate * daily eeey for =iee aepenh: at.. the machine based on aoe 


— Niet fhe best, (Shortest) response time’ pdssible. Too eagle 


possibly because: it .was’ ‘not known. whee oe 
response time.was needed, This determinatidn is. most’ pry neve 2 


ve computing - systems, surpnisingly ‘Vittle. 
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Pea. me a eis Fe . 7 rm ' Les Pe t . . ae ae a ‘go 
Pe if from many different data entry jobs ‘at several different keypunch 
ps os yinsballations) (Klemmer,,. 1960). No consistent’ differences are aoe 
oo yt Peported: between alphanumeric and Straight. numeric keypunghing.... + , ep? 
‘ Ne as a ae . Dae re a ee Sa 2 . a ae Pe 4 : 
i’ air For tnput.to interactive computer systems, the data most, | w+. 
, “a widaly clroulated:is hased’ on’ users of terminals constrained to a. 
.. ov. rélathvely low mag on Petes oF 10 or 15 characters per second. « 
- 1 "While the models - f terminals used in the sample -were-not.. Cae 
Dias explicitly identified, it may be inferred from. the date of the: 7" 
owe tele: typeuritens: mend the speed of tire ‘tefminals’ thet they were. — 
bet es sbele- typewriters, most likely. Médel 33: or 37 Teletypes. -Such 9.0. 
he Birals havea keyboard ‘layout. similar to that of office’. = 
byPewriters but quite differént:.in- key type and’ "feel". from : % 


Oo hapr cofventtonal typewriterd., mae ft Be 


oe . 2 yFrom- the: data-on high+volume typing given’ above,:it can be. =», 
hs seen that experienced typists. can often: exceed: the capacity of.. 

J. 4 thet terminal 'to accept keystrokes, especially for short bursts of. 4 
a er data-entry, This: data Should not~-be applied blindly,. since,: he 
a aes Just noted, the keyboard "feel" of a teletypewriter is not the - #7. 6 
"same as a ‘conventional typewriter. «and does,’in ‘fact, reduce the *.. 

J es Oo achievable maximum typing rate of theSoperator.- In addition; the. . 
Se _ loading of, ‘the ‘computer. system, ahd..consequently its-failure to... 
os . Reknowledge. input within “the required time (see Section 2..1.3) ors". 

sf a ~ the /limited size of ‘its buffers: for yet unprocessed: input hee 
"characters, may also serve to-Constrain the maximum possible rate 
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an activity totally unrelated ‘to network usage also +... 4 
a for an-interactive system to "crash" -at Any ota uring. 
- a. ‘conversation, ‘Such events produce distorting | data. To... 2 
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a single conversation,” where’ the upper limit is the entire data - Wg 
‘base obtained’ by the: NMM.. For example, -when all..of the usage ‘of... ; 


a given. network : during a: ‘period: of ‘time such as.a month.is: ks 


considered’, Jone ‘set Pnconpaees multiple conversations.’ ye age 


i tee thee phesent’ dontext the interest: ‘igs “ih sets op itch. 


"encompass less than’ a.conversation. Message! groups: may, be i ~: 


\". eonstitute | a’ subset.- There 


identified: according to the - functional “objective with which ‘they? 
are. associated. For | example, inva programmi env Lronment. the: ~ 
use of the various ‘language tianslators, | the" ditor, % the linking. | 
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“J, Yecorded.on the. following n&pber of days in-each of the three . 


or a ee ee Oe he a 23 
ao a ee Se oe 8% “Sewell ITO Be days . . oe re ee at (3: 
at Ta ee Table 3+2. Summidny of Data Recording an a 
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: : Ale a rsa Se Fi Ain ie oa 
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eed had been... * ie 


. : Dt ‘ : Be : é ue 3 . 1 i. 4 vos ie Sie 
vortial: ‘interactive sessions.* tHe LeelG of acoedtabue: a 
_conversations ta total conversations recorded was. remarkably ioe 
; _ ..zelonly about .one in three conversations was deemed ‘suitable for 
i wef ar Wer spi opeestna, Many conversations were discarded’ because 438 
- the user: failed totinteract properly wi tp the system, e.g.,' Short | 

conversations consisting. entirely of. eee -and: thus. on a # 
unsuccessful Login attempts... ¥e 6? i ee” E _ oo 


The result of ‘thig-cdlling was’ vohe ‘following, nudes of 
"good" ‘and "representative", conver sautons. for each. of the 
_ calendar yee and ic ated: . 


it~ tg fete eo 4974 
“oe ge ee OG, OTS 
wl Diy 0s Total 


- 78 conversations. oo ey 
‘119 conversations _ a or ae 
86 convérsations: -- yw *% 9 
Eee sco vGt ee tena: ; ne. 


4 or ae 
‘Table 3- a Tabulation of "Gooa" Conversations. by. Year ee 
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=> . Following the. emeeustne to pit the conversation data: to bie 

SAR model, the speed of the conversation and the ‘duration; -both - 

_ in’ terms: of time and number of message groups, was available. - 
-. Table 3-4 presents the results Of. data: ee tabulated. , 
‘terminal. ‘speeds. ie ed ! 
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an 


oe 8 Grouping: by ‘bernitnale speed: was. ‘ac oomplished ‘by rere rrr eg, 
the statistics for. entire conversations identified’ as occuring. at. 
the same Speed. ‘The. results of: a the data. pruned: in: , Se 


- Dt. as ne: lower Limit: or 10° message groups to! fan aocenusrex 2 

hy: converspttot is. admittedly ‘somewhat. arbitrary , but was: chosen ~ 

“ysafter® ‘8 amination | of? ‘a clarge numberof brief convérsattions — 
revealed most “tobe. representative Of error. Bend ations: rather 


this way “are discussed in. Snape oe : ; is 
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a _ “bhan formal a a . ae oe — s . oe 
Ep ie Date of; ‘such ofisuccessful login ‘atbenees may be at eree a 2 
oe cand. useful dn the. study of the ease of inter tive system use. 
OSS Mowevers: this Was-not “the~ ond eee of. ane “prest 1t-“Study.so hese -. 
pe ee ete AG babe rated. | i ae a ae ee ee 
eee i a Sie Bee a aA : wee 
pas a = A ' 3 ge, AF. : ry ‘ 7 ae ; . 
PERC he Mlb areca ie. ee 
Cem . bes , Aaa ook 
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bps data was’ assigned ta’ subset8 ‘and so’ analyzed. . The results of: 
"this AST OUp ie and ane 


Namber of. message groups: 
: otal time (minutes) : 
A" “gable 3-4; Tabulatio 


a 2... - oP eae : ; 7 7 : a - fe . 
3. 3. 2. Subset” kat 3nment~- ee = i on “ae ett ae et ane 
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ae 3 110 as ‘150 * ee coe , a 


eee ee. ee ie a 
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oer) yes 
tt e nature of subsets’ was ‘discussed if Section 3 54, fh. 3 es 


the: Univac 1108, subsets may’ be assigned: automat leaii) by 3 
program which scans for the- “master. space": character (@). ‘that en 
. Degins.jal1- executiverdevel control. statements. Such a program a ee 
was written by a: research assistant™under’ the author’ 8. direction. °. "ee 
‘A different Peay ce defined for each distinctive class. of: 
t 


‘executive’ lever Leinart atid that: at item, Peueee tee eae a 
58 tang patio 


ws massage. groups. up : ‘to the next: reodgniz d= tive'Teve 


Statement, ane’ assigned to ‘thes subset, 


] The brduping by application’ ts: oe by" dese 8 a 


a tistices-for all message groups assigned. to.the sarfe subse 


Due: to the relative paucity of data at other speeds, ‘only .thd™300 i 


vee As. eee in Sbapter 6. 


ee 4, 0 DATA ANALYSIS 


. “In: Chapter A we euclaned the supevinentel ‘aie sonnet hae es 
. data collection process, -the quantities of data colleoted and the. 
Poe. paste: data analysis that was considered part of the data eollec~ tc 
or “tion process (viz. :the application of. the SAR model), In this | 
i ."" ehagter.,, ‘we discuss. the statVstical treatment of the data by 
‘collected, including the experimental. design for data analysis: | 
.. and ‘tests "of. significance, In succeeding chapters we will § ._ 
- ddiseuss the résults of the application of these.data ane, 
7 procedures for a pumber. of ‘different ane pretene: 


ML Review. Of Data collected . 


a As explained’ in the: preuiecs ‘ve! ‘hapten és the” basic events 
‘recorded by the measurement instrument are the souree, code .and 
~time.tag for each character transmitted ‘by either the user or the. 
system; Groups of. characters.are fit to the SAR model, pesultine. 
or J, - in a ‘series of ‘stimulus~acknowledgement-response triplets for: “ 
seach conversation. The statistics. available for analysts are. ‘in. ” 


+. terms of message groups, not. individual ‘characterg, ‘Four-teen® ss iy ee 

“i. para ate aré. computed for each message group: as. summarized. An: me ae 

7. Tabl@.get.: Thus, ‘for each conversation, the available data. 
ao “.-eonsis s of a matrix of.dimension 14 x Ny. wher N isthe. number 

ae of. message groupe: ‘in that. eon er eehnOn rye 

~ Poraneter . \ Type wo Deseription —_— . a 7g See 
ee D> Pe, Basic * ‘ Re sponse- stimulus delay.. (think) time” .— 
78: '.. Basic. ‘Stimulus transmit time. | ee as ae ee 
—co° * Basic  , Stimulus character’ count © ~' ss re 
eh -ane 3s oe - Deriyed “Stimulus” ‘tr an’smissjon rate © ek ar 
Spe ae og Basigy. .. Stimulus-acknowledgement delay. time - ee? 
Tae a Basic . +: feknowledgement transmit time qo. 9 fe oe 
pei ee ree ' .. Basic. - * Acknowledgement character. count .: 

Te eg Ge oe Be leh Derived }. Acknowledgement transmission rate. -. i Rg 
So th oe ee i ‘Basic. Serre eine Gauge aes time: tae ee 
ee ee ae "7! Baste Response transmit. tim ae ee 

_ ae ae - Basie, *.-- Responsé character count: ee wae 
4 eae RE Poet Oe Pen rae ate: eet te 
oe OS Dee <a 8 Derived: Stimulus-response dela time Gs a aise 
ee Co te Derived. _ Stimulus inter- arrival ‘time » be, om AE OEE 
a ere oA Table 4. ae -bapnery 0 of Message Group Parameters. 
aa ge gs PPS fe gaa 
j ie. Male Possible. Groupings GPa OU e. _ 7 7 4 a oo 


a Compound analyses i ding CU eo by aggregating the rs 
for all the message gfoups in av~single or éroup of conversatian 
“or by aggregating selected message groups in a single gr reek 
poser ee These aie ie ane summar 1zed. bei lt in Tab 
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. CONVERSATIONS © es ct a 

a . | Behavior of | 
| ane users... 


. Behavior. of ~ 
-. One user on .. 
“selected tasks \ | 


- Single . 7 


Typical behavio 
of all-users-:onf 
selected tasks 


Typical-:behavior: 
Of:all users. 
(interactive. 
workload). -. 


Multiple 


ae a er 
‘The behavior of: individual users is not the condéPmof this 
| Study;. hence, the analysis of ‘sihgle corversations, either” ete a 
- entirely or in part, has not-been undertaken.. The jntent of this r rat 
—f 9 Study is to investigate the typical behavior of groups of users '- ae 
are an' general and on. specific: tasks, and the performance of the. - | 
- System to such-typical demands. ‘Thus the types of afialyses to ° . fo 
-. be described in subsequent shapters,chil into. one of the botten in ee 
3 _two cells in Table W-2,0 00 i ee ii ee oa. as Pg ee 


a a ay 


Table 4-2.’ Datta Analysis: Alterhativés. 
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‘4.1.2. Data Independence ~ .--' gt es 
eee _ One question that needs to-b addressed whenever data are. 
'. batched is whether the’ individual lobservations aréindererdens of 4 
-' one. another, . This question is particularly important in the case 
a pee ei dine ted ey such as we have in tha sequence 
_. of message groups ‘comprising an individual coriversation,, If they. 
_ . \:observations prove to. Be: highly ‘correlated in'a.serial fashion,. 6°... 
fo then we will. riot ee justified. in aggregating. them as:individual .,. # ne 
ae “observations with data from other ‘conversations. - ’« as a Soe 
i A a ee Rae ire phe 
'. ). The. questtoMof independence in. a titre series can.be- oe a ee ( 
uy, investigated. by means of the autocorrelation or. lag-1 dorrelation: = 7 
* “coefficient which provides.a measure of .the-overall correlation Pie eee We 
.. between cagh Successive pair of observations. This statistic -is\ goes 
. computed a ‘follows (fora series of .N observations): ee a eo. : 
food. : 
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valere X is tie. mean of the x i 2. re 
sand: oe ts the ‘standard’ aie of the Hi a ao 
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ee ek Mamyak and’ DeRuyter C977) suggest. that "a: reasonable. Rube 
a Of ‘thumb/ fpr deciding if the data are indépendant" is to take . 7 g * 
TET Tp re as EVITENCE Of no correlation: | ‘They further suggest ©~ 7.” 


“batehin of data wherd' volume ‘permits: Ce, ge taking. the mean‘of °° ~ 
,- - -every. njdata items asthe elementary datup) as a way: of Os, fs te Ae 


ae veliming ing autocorrelation. ee ee fe pes ata, ae 


b1ebioai. techniques exist ie hea ae the probable ee 
2e jofautocorrelation within a-single sample such'as the p. A ee 
give message’ groups in a conversation. - However, ina large. - 
‘samples when, for example, the autocorrelation sae 
1s. ‘tes éd to: be less than 0.1 to..some confidence limit, we would. 
still xpect some samples’ to fail” the test purely due. to random. 
 variat on. If. the degree of confidence for the, test were chosen. 
to be..95%, we would expect 5 out of every 100." 'sAmples drawn: from: 
HO a Beh UL uncorelated. underlying population to fail the test. | 
es 7 - Considering’ that thg number of samples in’ this study is in ‘the - 
ee es hundreds, a procedure is required to assess ‘the degree -of- auto~ 3. 
a correl: tion for. each: {parameter over. the. complete. set. a eenpies ee 


on so @ 
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fe way- to ‘do - this. a to. Soueicer. ‘bhe- erator of oe en 
ese! relation coefficients*for:all conversations, A Somputer . 
. program ‘was writen under the author’ } direction. to. compute Py. 


for all: fourteen. parameters. for. all- conversations, - If the data." - 
‘for wach: parameter are not ‘serially. correlated, we might ONDER as 
Pp, to “be normally distributed | argund. a.mean of zero... To... 0) | 


determine this f the: values of P, for. each parameter were ts he _, 


“protted’ in relative frequency histograms - (Figures Yajima through. * 3 
Yemjen)..\ Even without formal tests of central tendency, it canbe . i 
crpdad{iy eer that mest system-relatad’ parameters. do. ndeed- have 


| a zero as ‘the: central" tendency. for, A. \while phe. user- related 3”, sneer 
- “( stimulus%. parabens neve some : doderete d. Bree. of positive. ¥. 
es aurogne re rav ion: ia, cc / 


o. For. ‘tho se. parameters ‘vhare surcooneration io nob ‘evident, 
ae thes date collected: can safely be use Howayer , - ‘for =: eo? 
arafheters: where autocorrelation’ is- ev.id nt, use of all:tthe: ‘data. ‘es. 
aor yiell” pisleading results;,Since all\the observations would” woos 
not’ be—truly independent... For these:cases>:a compression of one 
ola “is neauhrey: to:. obtain ‘independent observations. -. 


= “One. method : ‘sug’ ere by: ‘Mamrak and ‘DeRuyter (1977) for ; 

— eliminating. ‘autoco ‘elation.is to batch the data (e.g.:, take. the. 7 7 
esos mean of every on data-items as the elementary datum). . They. : cee 

: yo vobserve tha ee batch, size: (n). can be-increased. (whgre volume: eet 
to pee untd an pecs ouablcr 
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nable to. 
~@xpeot s@arial correlation between the message groups’ in each | E 
eS a | eae conversation, but not? across conversations, For this. 
aU). reason; each conversation represents a cdivenient. batch for the 9 °° 
Pa ape «set of message groups within it: ‘The mean is still a A a) er 
wee _, iMappropriate choice of test Statistflc, since it is | highly. 

ar 2 influenced -by outliers, so that the median can be chosen: agair-ag—-—-.——! 
or ~a~bettertest “statistic, “Thus, by forming-the distributions of °.. 
- ;, parameter. medians for @ach conversation in a-terminal Speed. +... 
¢lass,’the same statistical tests used for all data can be . - 
~ applied wit 
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without. violating the assumption of independent eee he 
observations, However., the sample size is greatly reduced when. 

_ this is-done, so that’the lb pads e levels of the tests will 

- not be as great as with all thd observations: used; ee ee 
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gee eS Inspection of the distributions pbtained- empirica ly for <a 
= ~ooseach Of the fourteen parameters revedl s that none of then eppearg 
_:. to becdistributed normaly .° Thus,- the parametric ests*of — |. 
. Significance that -are ordinarily ae ee usq@i hat 
. directly, “'It‘would bé possible to seek a transformati l that os | 
“would result in a normal distribution; however, if tests can be .°- . 9 
-found.-that work: adequately without the necessity for such a ee a 


“transformation, they would’ certainly be preferable. ‘Two =. 


candidate tests are. the Mann-Wiitney U-test and the Kolmdgorov- ens 
Smirnov. test, neither.“of which dépend on any assumptions about: < 
~ ‘the shape of the underlying distribution (i.e., they are non- °° ° * 
oy DEVE CEE IO ECSte ee ey ee . 
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yg 4e2.4 Mann-Whitney. Ue a. a ee ee ec 
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fo iene at Least” ordinal meaguremént™has been. achieved, the-*- 


-. Mann-Whitney Ut St may. be used to test whether ‘two independent, .. ee 
' Broups' have been{drawn froni the same population. ‘According sto. i. 
: + Slegel (1956), ¢ is 4s one. of. the most powerful of thé non- Le eo ee 
‘. :.  . ‘parametric tests, and "it -is a most useful. alternative oR Le ee 
ie os  paranmet qosunptionss; ener “eseMrener wished. to avold. theo Myo 
t-tests: SSumptions..y" 0 sD es 2 a . ate Lee eae 


are oe ‘The: Mann-Whitney U-test is quite: simple irk con ®pt- and easy “/ 
aie :.f0 perform. The null hypothesis jis that, given’ sambles from tue: °° 
er . populdtions, the populations havé the same distribution, ne 
ot Alternat: hypotheses. can either be one-tailed (directional)or o 
Po bwoetall d. To’ test the hypothesis, the aeicva and" « 
a _tanked in inereasing order. The test statistic -U ds given by the. 7. 
. _ total number of times» that a scor® in: the Sample. group with fewer © 4°... 
.)  . observations preceeds a score in the group with the larger number“. .**., 
boa of observations. ‘: Th “contribution of Mann and Whitney. (1947) was. 
 * .* tO show that for Large sample sizes (520) thé-distr 
"rapidly approaches the normal®distribution, with .\* oe, F. 
teen i eat eee ge aie, ee OO ag eke 
a a 7 ae oe oe ee 
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ya Ang, ; Hs ges 
+ (which re a eee nowmally savinont it zero mean. and. unit 
—wardance.. - f. . ’ ; re 
ee A Capea eee performing: the ynocunseney U- bast is. 
_ provided in the IBM Scientific’ Subrou ine Package (IBM, 1968). 
A program was written by the author to permit selected data. 
.. eQllected by the Network | easurement. System. to be tested by - pe 
“i Mann-Whitney. procedure. using the..subroutine. The values of: U, 
Poe. cand’ the associated probability are printed. whenever the. proeran ae 
ae run ‘fi Ya setfor data. This’ Beceem. was. used. in the analyses. 
eg ae. 0 by deg ribed n Chapter. 5. : or ees ee a rr 
\. ee tee ie y eae. . a ee ee a : 
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ow, 2; 2. ‘Kolmogorov- ~Smirnov Test =. 


oe | Bea The Ko lmogdrov- Smirndv eiosaanp le: test, is another bast of” 
an whether ‘two independant : ‘Samples have’ bieen drawn fromthe same. 

i: _ population (or from: populations with ‘the ‘same distribution) bat 
. or .. (Smirnov, 1948); “The-two-tailed test. is. sensitive to any Lge of: 
“.. ..° difference in. the distributions from which the two ‘samples 

-drawn, e.g., differences im.eentral tendency, dispersion, higher | 
“moments. The one-tailed: test is used to decide whether. or not -_ 
~ ‘the values of: the population ‘from which one of the. samples was - 

drawn. ‘are st chastically larger’ than the- values oF. the. epee 
from’ which. wl e other ‘sample was. drawn... 


as eS . a 
ae oe The. actual: test. if seneevned: cwith “the. Bheedent’ borwecn bie” 
” # “ountulative. distributions. Ifthe: two: sampi¥s have..been drawn... - 
A _ from: the sane population. distribution, then the. cumulative. aes 
wt. distributions of. both samples. may be expected to be fairly eae 
-. to. each: other, ‘ginte: they. both. shogild .show only random’ deviations 
‘. * from, the population distribution, - ‘If the two- sample- cumulative: 
. distributions are “too, far apart” (Siegel, 1956). at any. point, 


his. indicates that the- ‘samplés ‘1fkely. ‘cam’ ‘from. different Mr e 
pophlations. Thus, a large. enough deviation: between .the twor we 


Lhe ; ve qunulatéy distributions is evidence for rejecting - He 
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_ eee. from a ‘the two Samples were taken, 


py : *. Lea aa aor (20), 
ae me Subroutine to ee re fhe, Oladdorove Smirnov test is. also aie 
“provided in. the IBM Sciéntii Subroutine Package, and the author °" 


mals of data’collepted by’ 


a 2. 29/Manking ‘and Selegtion Techni: ues. a 


2 "ype er formance of. some number of eee ee of, observations aes o 2s. 
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“/Select \the "best" wiggh some level of confidence, then a rege 
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“null -hypothesis) of e 


prepared a program: to“utiliz this, subroutine for the. analysis Of. 
ee Méasurement System. he is 
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data. This Br oe ren: was eS used ae 
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baséd on sample. means, 
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atty on the work of. Sopel. sas 
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ae the pee ne and their applicability have ee ecome | 
! available (Gibbons, -OlKin, and Sobel, 1977). .When appligd.to 
 Selgction problems, the procedures. specify. thé number “data. 
points which must be collected as each alternative if order to 


-as\great-ge.8 some ee Le ae 


‘}son\studies,.using. data eet lected with. Cee. 
remen fy tool: degeribed in the previous chaptér (Mamrak and 
er 21977; . Amer’ ahd Mamrak, 1978; .Mamrak and Amer ,"-1978). 


present. stud (and. with. some-contact- ‘between the researchers). 
However, the. purpose of sthese studies was:much more specific 
ca (aimed at: ‘procedures. for ‘computer. selection) - than theseurrent 


cling: ‘in gener 
o dppropriate for use. 
pay os Lan and now suf 


). Thus, these. procedures were not 
n the current study, ‘though. they. ST 


‘Buarantee tha@ the probability. a-correct selection/is as oe 


In a real sense, this work. has: been gong on ,in parallel with the 3 


elently well described (afd. withthe: Cs . 
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eee _5,0 EFFECTS OF DIFFERENCES IN TERMINAL SPEED. ee ds 


7 |” Previous’ chaptetts have dealt. withthe Motivation for this: : .- 
research, -the methodology for..data cof¥lection and the: data = | fie ae 
“analysis Pip argures' In.this chapter! we discuss the results of. 
“ - -dnalyzing: the dafa according to terminal speed, |- This. serves to ae 
“.“ spro¥ide onedéxample of phe methodology as well as.to obtain Some .- °- |. 
oe .. practical ‘results, viz, descriptive data aboutlthe performance. :, ~~ - 
et OE users ‘(at terminal’ rates higher. than most: studies preyiously 
/\o. Sreported) and the -perforiince: of ‘s representative interactive.... . 
: 7 yA WE eat : : . ‘ ~ Fe , ae . 
- '. °. . AS was. explai ed ‘previously, data were collected over'a we 
\ a three year period fot. interactive ee of the Univac 1108 system |. 
\. at the National. Bureau of Standards, -.Dué tothe availability of. | _ 
|. . a,Wide variety of terminals during .that.time pertod,® data WEP@: ey OP Se 
so") collected: for users operating terminals at. 110,’ 150 and 300. bits 4 - 
i. per second (10,. 15) and 30 charactérs per second), The data... _ - 
ae collected.were aggregated according. to terminal transmission rate - |, 
_ “.and- analyzed to isolate the effects of varying the terminal ra@e .- 
-Qh-user performance and on system performance (as evidenced by,« - 
|, the*parameters of¢the SAR moded), tye a a a ee ee ee 
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oo. Experimental Design = gee Oe ae ie re er 2 a 


| | | ‘ 2 a ee =). —— 
oo The basic experimental design forthis chapter is to compare. 
. the- effects of different. ‘treatments (tépminal speed) according to |. 
.. a number of parameters (the Hi par anetdes or the SAR model), ee 4 
” Actually, the distribution of parameter valu Ss for each treatment... - 
.. Broup will be compared for each parameter ; me null hypothesis, 
. Hj» is that. there is no difference in. the distributions, i> ae 
oe that the. samples were-actually drawn from the. same. populatio Gna date : 
woe ied onc paraner cic tests discussed: in Chapter 4iwill be used *to” -— 
"oe yo test thé significance of observed differences .in' the ee a 
- . |" distributions, * Although there weré actually three different 
. terminal: speeds for which observations#were collected, ‘s0 few . 
data were collected at the. 150 bps rate that these data are not —. Se 
hemor _ tested, +(Many:of. the results. for. the 150 bps data-appear to. be ac 
5 @.°) anomolous due. to..the relgtively low numberof observations); ~ eS 
* Sfhus, only a. pairwise test. is performed. comparing -each of the ok ar 
: treatments '@1%0 bps versus: 300 bps) on. each Of the 14. parameters, ©. ° 
‘using -bobh the Mann-Whitney U-Test. and the. Kolmogofov-Smirnoy. |. oe 
“ee Test,» These: tests ‘are applied both -for the distributions of all: 5, ab a 
’ ethe observations and for the. distributions of conversation a St eee 
“cvs qmedians, “as was discussed in’ section Wade Qe Oe eel Cahn 
“ot a ee ee ee On ee 
a ae All of the usable-‘data ee age in ‘th@ study are’ analyzed’ ~~ . 
i in’ this’ chapter, A summany.of*the nugber o -"goad".’cdriversatiéns: ‘. . .: 
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: In.each ‘such set of conversdtionsimaybe fond ‘in Table .3#3. °° | 
However, there abe’ not as inany daté items fpr each’ parameter as 
the number ‘of message. groups would indjcate.” Message groups in ; ee Seg 
cagty conversation (particularly. those which initiate or terminate, oy 


a recorded. at each data rate andthe total number of message @roupe:: _ og 
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“tn the tol Fowens fara the- “Airiee observed. Valu 
Las ef these. fourteen parametérs: ‘are. presented, both? An v oe % 
graphical ‘and, abbreviated . tabular ‘form, a8 cumil agive : frequency’. 2 
ice “da stributions ; It ie felt that.this is’ the most meaningful” Wey 
pace whick to present “and. ‘inter ret: the regults, ‘since the’ ondinfany -. 
frequenc yhistogram for all-of. these parameters: is decided] y: 
one normal. Thug, as just d 'scusséd in’ Chapter: “HySnon« ee 

a ests ar used to compare differences epee distributiohs. ne oe fa 
“2 These. tegts themselves are, based on, the bse’ of cumulative Sh ae 

<< & 2 distributions, “Prasenting, cumukative- information also mininizes. o 
f he &ffeet “of: outliers, either ‘at the. ley.o on ae ‘end of the rs : 
petra is function. 0 ae <_ @, 
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When. hese. parafeters. are: “uged ars oa (ee ey ane - ‘ » ae ghee 
ge aide ‘design. goals), certain: points on the distribstion a ee 
Pequ: g bo gertz in eumulative: percentiles) are. typically *‘e ar ee a. 
fees hed 20 The most’ gemnonly. specified of: these, the median’or’  . oo...) 

50 ‘pencentil e and: the. nigel presenta are tabulated ne, Le eae 
ee from, -bhe. comp ete. eraphi al presemtetiony.. 7 ie ee 
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a ce mat ‘adequate’ number “OF observations has been obtained to: ce ee eee 
a oe ae “Statistically ‘compare, the distributions of parameters for. - ee 
pel an Se terminals sat : -the,110.-bps and 300° bps:rates. The mumber of | - eae 

Py ofservAttons for the 150:bps tarminats does not appear to be:) 9 4.) 
radequate | if. hence Wtre se * data area not analyzed and aré presented a ee 
\ For Rieke aa Many. OF, these oe SppePiert me pee 
poe : _~ - aac a eee | aes 
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iY gitterbnees” 
‘parameters Aw each case. ae 
yethough the. tests’ results 
@- following ‘sections, the - | 
Se ak cussed “ina separate /< 
‘agul ts * ‘havé- been: discussed.” 
tiethe Martin-Whitney U-test and 


ppl ied’to: compare the Hee - 


a ach i dade , ‘all the differences between 
sigtiritant at-the 14%*level. or bette 
ere! of sertal scorrel&tion was noted for:some 
; "Siok n: Section. 4.1.2, When the-data aré batched so" that: a 
“e singte Statistic. is used for.each’ conversation (the - median. of . 
each: parameter) . ‘in: order to eliminate the- serdal' Conrelation, che 
“differences: between. the distributions. fora. few wes ane | 
“net. St ghi Tie an ats even | ‘the: 54. Jevels vos A 
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The’ user: pevtopmance: pennies are those” ‘Bevociated with: 
timutug - ‘Portion of the message grou These. parameters’ ne 

budéé.. response4+st-imulus delay Litke: (also called "think" time) js 

>iLmuLus transmit: times stimulus character count and stimulus ae 
“transmission t rdte.. What is: Qf interest'here is to-ascertain the: 
LPP WOtL ON the user (changes in wser: performance as evidenced by 
the ‘effector the four. parameters. descriptive of the. stimylus | 
‘portion. of the: message eroND) of: an iucrease in the operate 
. SPEat: oo ye ear : 


Ree “ rt oe 
a fesponge-Stimulug Delay (Think), Time “ 


-The® ‘Hesponse- Stimulus delay t-ime (ps) is the’ time. in os 
seconds. betweén.the Lhst character in. the response, portion. Offa oa 
‘message ‘group N and the first character in the «stimulus portion: 
‘Of message group Nel. “Tt. ts) characterized as. the "think". tim a = 
“since it represents the latency. period during which.the’ usery., 

"digests" the resporse. to. sei * ‘previous a a) formul at S. 
ERE nex te stimulus, 9 oe — 


= Measurements of Responseé- Stimulus delay time: fer. all néasagp” 
groups, ¢ ategoni zed by terminal speed, were distributed ag shown. 


tn Figure.5-1:5 The: key. ee valu eh 
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ee Number of observations as given in Table 2-3, 26 ak t 
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Cees ee ee ee te aye a ; 
, |” Operating Speed of the ‘termindy fs inoreased. Since the interval oe 
he ta for user think time does not begin‘ until. the Yast character of #7." r 
Joo dene response (trom the previous messager ereco} rem eee Sent, ws: 
oo. this reductton can not be attributed to“the ability of the: 4” 
_. Operagor ‘to-read’ ahead while: the response ‘is being printed. If - 
anythin, reading aflead. would tend to reduce the think time’on. — 
J ckower speed termina: Since the, text of ‘the response is =: |: 
—.Wisptayed: forta.londer riod-of time before the think time. 
interval begins. Thus, the reduction-in user think. time must: be 
attribute to a -genéral increase inthe pace of opérator, actions. Ao 
2.0%. PLied bed, y hs Oppurtunity..¢g interact with the.computer at a 
DER NOr rake “ea ° | me % ae a8 oo a ee ee 
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me, a Oo ee pe anh Sie ey a 
eee ne Re ee stimulus, Transmit Time =. . 6 oe ee fo. ee a ee 
eh > The “stimylus’ transmit “time (TS) is the time in secondd . | . 
/ rem thee first ‘te. the last character’ of the. stimulus. THis °°... 
“parameter. refelets Both, the user rate of data entry and the: oor 
. Wamber of characghers entered. By itself, this-parameter is AN 2 SaaS 
_ > ind ieatfon-of thd transmission time required bygthe operator. on”. a a 

_ the channel from (the. termirtal to .the computer stem, The” ob ne 

@ stimulus. character count (parameter. 3) is divided. by the Stimujus: - a, 
i transmit’ time ‘to’ compute the operator. input Or typing rate, > ow. 8, 


_ Mersurements fof stimulus transmit time for all message oe te 

PPoup? , enbegorized by terminal Speed, were distributed as. shown, — sa 
J LU FL gure cee. The key. percentile values for each class are’. et ge. 
» , babulated-in Table 5-3, both’ for the distribution of all. | .. & ae 
we Be observations and for the distribution of conversati nmedians, - | | 
ee “The gifferences: between the, distributions form the 110. bps and the 
te 6 S001 ee are significant ‘atthe 1% level, both with and 2 + 
Without batching of the data (see section 5.5%, 00 0 00° bo 
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“ ~ character count with increased terminal speéd: would tend to 


evident in-Figure 4-1-b)... When the ‘serial: correlatddén is. 
, » elflminate@ by taking the dist ‘ibutions of. conversation median 
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“for. al; data as the terminal speed is.‘inereased from.110 to 300.. 


-bps..(the value for 150 bps ds ntempreted as béing ere his 
“The 90 percentile values are about the.same.. However, 
~ parameter exhibited a certain ° egree, of: serial correlation: a 


different picture emerges; stimulus transmit time, ee ‘Seen to. 


this 


s~ 


y ees 


_decrdase ‘as the terminal peed: paoreaseay io we 


The change in. stimulus transmit” time. ag: a function ‘pf - 


working’ in opposjte dir etions. Any increase. An timulus 


terminal: speed répresents the net effect of two oe 


Imetease: ‘the stimulus transmit SANG while any. ‘increase in “ 


ad ! ok i ee 


, O 


2 “stimulas7€ransmisatoni ¢ a would: tend to decrease stimulus: ee ae 
20)" transmit time, Both of these tendendies are evident: to. some. ee, ae 
0 0 degree (see the following two" ‘Seations); ‘apparently, the ° 


> Inerease in ‘stimulus t ansmht'ssion. rate more than makes up. for: any - oS 
Le oe oe an’ stimulus. Q Maraoten fount. bac * eo ~~ Sy 
: 2 233° Stimulus Character Count a hit Bg, ete | ae 8 ; ce 
oes ~The. stimulus ‘chargoter’. count (C8) is che number. Of ig, eee 
a cee Characters eobered ne the a ae input to the. ny a ae 
@t a system, ; a re oe / te A 


 Meaaupenents of. stimulus gharacter: eount for all iessage 


ae categorized: by terminal. ‘Speed, ‘were distributed as. shown. . 
in Figure 5-3. The key percentile values ‘for each class are: 
tabylated‘in*Table 5-4,:both for: the distribution of all. 9: = 

_ observations and. f r the. distribution of. conversation” medians. 
_.., The: differences between | ‘the ‘distributions for. ‘the: 110- bps and the. 
"300 bps ‘classes. are significant: at the 1% level. without: batching’ 
° Of the datas. however. they are not signifidart - at. san ice ae cc 
"level when sere * data. are: batched: (see Section 3. oy ee | ‘ eos 
a, oe cer. All Observations 7 “Conversation Medians mes 
“oe eee) ee eae we 35 Shee yee hae 
Poy fe ts 450 BPRS tae ee. 2. oe Se. -6 "8 — ee 
i 2 300. BPS - 6 o~ 23 - Bg aay . ft * 3 ne ig ae — 
‘Tabie Os 4, key, Percentiles ‘for ‘Stimulus Character gount. 7 eee a 3 
cee The. ingréase in ‘stimulus’ Shabaober ‘eount foreeiy oy Elie 
eee observation! that is associated with’ higher speed terminals is gt ag 
vo. 6 one Of the more ‘interesting results -from thé measurement of user >. 3s 
- + performance parameters. ~ Apparently, - ‘some users of higher. speed — 
., (7. terminals. feel freer to express themgelv, Ss. sonenhee more’. . o's 
a Nar eaner ys to, the: Pompuber cane de : ~ A rr a 
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«speed terminals, but this was ‘not explicitly tested. 


hagh ser Lal: correlation (see Figure..4-1-c), 8 
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Surprising . that. the distribution . of conversatfon: dians ‘do not . 


However, at mule character ‘count ese st particdwarly , 


Ju. “show the same. phenomenon. © Thus, we must conelude that the. trend 
’ <'.: Shown for all observations. is the: result of a relatively few. : 
‘oo. conversations-with consistently: higher stimulus character counts “3 


at the 300 bps data rate. It might be the casgg@hat users with — 


"applications: requiring longer ‘stimulu@ character, counts tefded. ‘tov 


have longer sessions on higher. Speed ‘terminals than ‘on lower” 


! ae A 


Aire O73, cuhulative erequgney ‘Distribution of stimus 


Tt’ Bas: subsequently. ieee wancs (gee ‘section: 5. as by that a 


-. number. of users were. inputting * papér tape on lower speed =< 
“terminals that. were. so, equipped, . This caused a dramatic. “change 


dn the results for stihulug Heli leen er SOUre PERE | 
. .@ligination.of: those. message groups re resenting Maper, tape . 
nol To. determine if eliminating these message groups had any 


atte tr on stimulus. character count, the data were trimmed ,at an 
upper limit of ‘first 40 characters and then 80 chatacters, This. 


was bagtd on the presumption ‘that papey’ tape input ¢eould: be. -. ae, 
kee cy ted by. the» Reneth, of one ‘tnput stream. In the first case ae 
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-” * wumber of observations for: 110: bps d va was reduced from 2638 tote 2 

“Pin 2572 and the 90% 9 pue- was réduced-fkom 21 ta 20. (The median... i" 

os) “". wascunehanged..) Wh! n'the data were ‘trimmed at 80, the pe hn 

. observations wae reduced. to 2617 with similar ‘effect on the’ a 

"median and 90% value.  Sincé-thie {rimming gs Shown to. haye kk me 
fon of al observatians, ‘it, wage. a 
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ae ‘groups, categorized by terminal: speed,’ were.distributed as shown. .- . 
eit Figure.5-4,° The key” percentile.values’ for éach nn ces 


aoa. qoeer ations. and for the distribution of conversation: 
“Sha “ « Note’: 

- characters’ per Second, in order that the extra bi ransmitted. ~. . 

je foreach character ‘dt-the 110 bps rate not distort comparisons of | 

f. user data entry rates, ‘The differences: between the distributions . 

for ‘the 1:10--bps-and he 300 bps classes are significant at the 1%. 

roof % 4 Level withowt batching of the data, and "they are Significant at. ° 

se be 54 level when thé data are batched (see section 5.5), though... 
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. 2 Upon examination, hese: resutts were’ ute ‘Sbartling,. since 
“.: they implied: user thping rates on low Speed termifials that are ae ee 
t intuitively unachievable. Further investigations led- to the. * at 
‘finding. that (1) many stimuli whose .transmission rate was at ‘the. oe 
maximum line rate (10, 15, or 30 cps) were single chara@ter 
ee (2). some users were inputting paper tape which. ge 4 ote 
te transmission rates atwor near the nfaximum of 10 cpg. Als ak ae 
‘the ,relatively. high degree of serial correlation, as noted in- a 
Figure 4~1-d, indicates that users entering extremely short | 
; stimuli-or Paper. tape: inputs. tended to do so throughout. ‘the 
cos comversation). : Thus,. while. the'user data entry rates: may have -— 
been distributed. as shown in Figure-5= =] and patulated in’ Table’ 
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» - . Pesults tor even their: statistical validity) at this beam | 

"2 “simply note. that the empirical ‘data collected during this sttdy. Bs. 2 
- "and: anal d “according to ‘application: reveal that.the user think: a oe 

(court: wer stimplus transmissidn rate, ‘and .the Stimulus character ... a 
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count: were all considerably less for XBASIC. than “for BASIC, while. 3 

- the, response character count ‘was. constderbly. greater. This might "| 
'. Suggest greater ease of use ang utility for XBASIC, ‘though be Ase 

, obviously a more detailed investigation would be required to. . : 

». confirm or deny *such interpretations. Hgweven, observed users, .. 

-, who. were. free to choose either interpreter, showed a clear... -\:. co 
ae preference for XBASIC as ihdicated by the mucH’ larger “incidence : ee 
‘in o» . Of observed message. groups -for It) + my ge ae oe 
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7.0, SUMMARY AND ‘CONCLUSIONS. , ee ee 


In this enaater ‘we review the. nesgits “of the abudy: as 


. deseribed. An the ‘previousgsix chapters. “No new. information* ‘ts. 
/ pregented, ‘but some cpne 1h 


sions based on: previously presented. 
information. are offered that heip ta tie. together some: of ‘the ' 
‘seemingly. disparate findings. »Areas 

results (particularly the methodology) are discussed, ‘and the. 


limitations »f the current study are eandidly- identified: The Ti By 
“dhapter. and the dissertatiop. concludes : with some. SuBpESTI ONS for © 


future work’ in this. area. —- .. —s 
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“TINA s. ‘study na eat both: with the deverspment of ao _ 
“methodology for: wthe quantitative. evaluation of ‘interactive ae 
comptiiting -and with he application | of that methodology ‘to a” 


of applicability for “the ws 


specific interactive system. In, the following: sections’ we review 


“the methodoloyy which is. ‘the most significant contribution of . 


this study. The’ empirical results from the,data collected on the 


particular system under test are also: summarized : These *esults 
‘are interesting dg. their own right, especiSily. when: compar eds wigh 


= _ Previously published. results, | 
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Th this dteacréstion: we have davelopdd. and: applied a new 


Bg for the measur efient and - evaluation. of. interactive. 
col 


puting. This méthodology may be applied to either the. users. 
fof an interactive. computing system (in which case the users .are 
considered: as, a system: component in the 


“approachy or to the interactive computin “ system, including the 


service computer and any* communications network through which the. 3 
servtce is delivered, The: ‘methodology may be used for a.) iriety | 


of ‘different appr eGald ons), some of which will. be'suggested in. 
-Section 742. below. Héte we review the methodology itself and. 
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eee ohe.: ‘methodological, contributions: of. the dissertation, - 


oolkect ton technology developed §t the National‘ Bureau of - 


The ebhodolopy is based a Tete eek ‘of a: aster 
Standards. As‘ explained , in Chap 


-can’be attached to, the data path: betwefn; the user ‘and the 


computing: system $0 that, it can identify and time tag apl - 


‘oharacters,flowing ‘in either direction: ~In this way, all ‘the: & 
' fundamettal data about interactive ‘conversations are: collected. 


_ ter, aubséquen}, analyses. 7 se. “see 
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rad dst fonal- human” factors 
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“ statisticg”’ computied 
'. Sets of message grou 
1 Conversations... Oe ee _ 


es As ‘explained i section 3. 4, ee tile muti hed: an atts | 
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- ‘model a 


" specified,. ‘data .limits, to ‘compute auto~ ~coptelation coefficients oe 
the distributions .of ..---- om 
. .two sets of any Single pkhraméter. to- library programs. for’ the, : hail sacl 
'; Mann-Whitney and Kolmogotov-Smirnov | Statistical . ‘tests. These re Fe a 8 
.additionaY routines: Provided the abies aoe, used: in: this’... -° 
- Study, .as described. in Chapter Ae a a ee 


~ available when és sjiaeareec sen ‘work rte was ‘the. oe. to ae 
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for these distributions, to trim the distributions above or ‘below. 


and form their distributions, and: to pa 


tae bot 


ay ” The ‘d data collection procedures and the. extent. of bie data’ ae se, 
-actually collected were. described tn Chapter | 3. -Briefly.,- data 


were collected on, randomly selected days and at. randomly: Selected ' 


_.. times Over a period of three: years, Recorded conversations were - 
‘culled to: identify those that Were representative of: "normal" Jae 
aa pper@bton op) the’ interactive System. | These’ conversations. were a 


then’ aggregated by terminal. speed and. the results analyzed. Two. 


A sets. of result; 


~ Mann= “Whitney. U- test ant Kolhogorov«Smirnoy, ee oe ar lc fa ts 


oo a Mnneater, pecondee. at: 300: bps: were also ‘Purt: er 


2 oe 


Sela “bo-the Lin 
‘Data were. collected fb 
“ee bps ends mee bps ..: 


S° were ipreeenved ae re >. for each paral r at 
the. SAR. model: te patie 


a 
t 


ary 


Qe T buLae presentation: of: ‘the distribution pe conyg trsation- 
. sa | medians: For. the. model Parameter. - eee he he . 
‘Both distributions (Marie. ‘and "medians" or "batehed) ne the 1105 
bps ‘and: 300. bps terminal classes. ‘were compared using’ the -].. 
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‘aggregated by: application Ns: and* ‘a brief. analysis. performed |: ore 
according/-to’ this" grouping » Acomplete.. table of results : or this” 
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. -observations was small) 


8 distributions ‘were: formed’ for. all the: observations: foe aun a 
parameter, and for the median: syalues within a eonveredy toy for 
“each: a 


cc. P : xs ? oar i a 2 ae 
~~ Logking first at the- bavameters eascatated: with: the be bape 
inputs to the: interactive system, an increase ‘in the “input 


st'lmulus. character. count was evident when all observations were. - 
° considered, -but\pot. when, the distributions of “conversation 
medians. were examined. §tnis ska the effects of.of high ~ 
os Serial. correlation for this: paramétef, and. indicated that a’ 


- Subset of users -terded to eave long systen.. inputs BhEOUEnaW Es, 
“their conversations. , moon ls re . t 7 
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& The user ‘rate of. data: entry (uaeee fae. measured from: when’ Ma te 


* the input began). ‘increased withthe. -fnorease: in - ‘terminal speed... 
“This. trend became clearest when a correction was made for paper ~ 


tape input and single character. inputs, which were transmitted at- —* 


the. maximum. data rate and. thus distorted’ the distributions. 
Comparison. with ‘previously repofted data (described im section 
"2.3.1)-,- particularly the Bell Laboratories! data (Jackson and > 
Stubbs, 1969) showed. See user data rate for NBS users’ at’ 110° 
‘. bps, a slightly lower a rate- at 150 bps (though, the: number of. 


ee any ‘reported. previously (though the previous study ‘did: not 


a plabte and) 


Pt ae 


oo of: oa terminal. Speed. 


- include. the 300.-bps. case); The: generally, higher‘user: data rates" 


folnd-in the present study may reflect, a more rapid’ pace of .— - 


 nteraction due to the extremely: good response of: the i eractive .. 


: » system. — This would be in keeping: with: the previous datd that. 
‘. showed -a oi ae user -data, nate on a’ nOvE, heavily: poe ‘system. 
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“was “found: to, decrease W 
-effeats® of. serial correlation in-the observations was ‘removed’ by 
batching), indicating “an| increase in| Pace. on ne Fe of: the oga% 
users: of higher, speed | le e 


*eokina® at. the computer output, the ae of the Aeaponues® 
“were ° ‘found to increase significantly | With terminal“speed, Thus, | 
“it is évident that terminal ‘speed: is a bottteneck. for\computer  . 
. output, and that usérs will adapt heir -behavior in terms of .the 
~ types ‘of: ue Gs they request according to the speed of the 


tetminal.us}d, . (We must conclude: that it is the: users, not the | ‘ ‘ 


“computer. sy t¢m, who are adapting to the speed-of,.thé terminal, 
-since the sdéftware providing’ any Economie 18° “ane, "same > regardless 
et : « 
oo. Lf we oan tweke ine ‘awauoe ion oe are more satisfied 
with ‘longer responses. (since they, request such de ore when. 
' they are able), we can mdke-an’ interesting ‘compprison aay fia 
“data reportéd: by. }- H. Milder..(deseribed jn. uses 
gn 


Miller fotfid no significant performance or: atte 


” The responsetstimut us. delay: as ‘of “Phe. user. "think# cae, a 
‘th ‘increased terminal speed (after, the | ve 
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and a‘higher-data rate at 300 bps. than oe 


age Monde 

-associated with an incredse in terminal ‘speed thon a to: 240° - 
rn ops. We found a significant inorease in- input: and output - Volume - 
(with a oe increase: a user haecsical ie vuser 
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Figure 7-1, User Satisfaction as a Function of Terminal, speed 
7 Response’ transmission-rate is one parameter for. which the” 
ooo, “mean, values are more enlightening than’ the'percentdles. Response @. 
ee _ transmission rate. clearly increases with terminal Speed; the.) 0s . 'y 
er ‘médian values geam to indicate that F@sponses were always ..@ 0: (+t *, 


0 transmitted atthe maximum rate, However, the mean valués. (from 


. represent opportund ies. to regain the “use of Otherwise wasted” 
transmission capacyty. through*some line Sharing technology such > — a 
tiplexing or: concentration. This study 5, a ee 
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se “20.00 Pherenoes. betiesn the distributions of all. observations of °: 2 . 
stimulus ‘character’ count and. response character ‘count forthe 170° ~.* . 
- bps class and. the, 300 bps classes wére Significant at the 1% ° beh 5 
‘level; flowever, the differences between. the distributions of 0° » 7%. 
Re . " Canversation medians for the same. parameters: and classes were not. 
uv... Siunifteant at ‘that level. . Stimulus charactér count: seemed ta ; . 
Loa, 7 @xbibit considerable ‘serial correlation (Figu -Het-c) ‘while 2 oe 
ae, os J, -Pesponse ghara ter count did fot (Figure 4=1. _—? a ee eT ‘ 
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eng ELANERY, slow transmission. ‘Tines.. | , | 
as Response’ delay Atimes were. found 6 ‘be atightiy vonesr: for. 
. higher ‘speed terminals; - “however, the. level of the system - 

\response to. the users*was so good‘ ‘in all cases, ° ‘and. the ao : 


oe highiights. the roaperuunieies for sharing that exist avg ‘on Me ee 


“tntmpor tant. 
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eo The scinoeestacacit portion of, jhe: message group wag algo - 


_s’\}nvestigation. This can be attributed- to. the relatively: Light. 
-- interactive load, On the: system during: the study period. During’ 
“..this period). 
/. + “being evaluated and’ access was ‘Limited ,to a/smatl: number te 
“Simul tanequs: users. In’ retrospect, it seems clears that. the 
: * - simpler S-R° model’ could: indeed ‘ have been’ used for-this- study ° 
oF withopt jnipairing the-results. ‘This in. no- way invalidates: the 
~ °° SAR,model, but rather, demodstrates that the added. detail cand 
| complexity) ds: oy taal ‘in all cases. |. 
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Chapter” 6 presented the results” ‘of: grouping. ‘the ° ‘data’ by user. 
ope prior to. analysis ‘Conly data ‘callected. for: 300 bps’ 


this analysis). ~The 50%. and .90% values ofall. parameters” are 
‘presented for all’ ete o aor a some. of the cen oe 
ae was” discussed, : ne A a 


aA Examples were. given of: two | yoes ‘of. analyses: eral vata: of 

— “par icular parameters and analysis. of particular subsets. . In the }- 

a oe vanalysis. of: particular. parameters, the.stimulus transmission rate 
os Vand) ‘stimulus> character. count - were examined’ for. all ‘application: ee 
mee subsets... The subsets were. sorted. by ‘stimulus transmission. rate, 


. -and then both, parameters” were displayed in tabular. form, This: |. 
ree “permitted easy recognition of which applications. had. ae 
_.., _scharacteristitally higher. onlower Stimulus: transmission rates.” 


Me ee ‘dt. was also easy to, notice. that “high” stimulus: 
: were. generally pences eed with, ‘shorter | ‘stimu ds: ‘Ghar acter: counts, : 
ig BOON vice versa. 


the anal oie of: ‘particular subsets,” te was “demofistrated 
imilar' language. processors (BASIC and: XBASIC were the. ~ 
“forthe. example): could be ‘e%}mpared quantitatively. 


for. the methodology; statistical testing: mas not. ee or 
ares two eas of. gare ta a ee Pa 


mindls was used ; so that terminal speed was not.a factor in Ae 
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ntent was. only to. illustrate additional applications |” 


‘the impact. ofvinteractive users on the, system . was. . 
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eee found. ‘tobe insignificant. for the interactive system. under”. see 


transmission rates... 
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tf rr ey Applicability Of Results A er re ey ; 
uae .. The eyner at approach that *has. been. developed: in this dis- 4 - 
,- “Sertation Should ‘be useful for a Variety of applications. I. -% 2. 
particular’, .the ability to test for significance of differences. - yo: 
_ between Sets of. ndn-normally distributed data is-essential if ob-.°— 
" / ',Jdedtive comparisons are to be. made between performance variables ".- 
>. ‘in. interactive computing, The following Sections present several __ 
‘examples of. possible ‘areas of application for the methodology, 9. °° 
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7.2.1 Procurement -— oe. a _ 
-, - . The ‘analysis techniques: which have been developed’ may be =| woo 
used’ in: the design, selection, pfocurement and improvement-of.. 9” ee 
_. computer systems and remotely. provided ‘computer Services. In the © «. 
+. design. of a computer’ system or network, the ability to méasure a oe 
 *-) external ‘performance 1s useful in specifying service’design goals. a 
/ . and-in: determining how well-*these" goals are met (Kamerman, 1969). 
/ . Objective measurement will.allow the direct comparison OL ge en acd 
ra ROW td eek eek epee eomeniters. and users alike. Thus, service... 
: } 0. Providers can impiement systems with Specific design. goals TOM oe eo 
Service,}and users: can specify,’ perhaps contrattually, the level’... 
“of .servi¢e they expect (Grubbe.and, Cotton, 1975)... ae ae a 
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seifiqation of the anticipated workload ‘is an essential - 
_ “pny: request. for ‘propogs#l (Ferrari, 1972).- Knowledge of* 
“current emand: provides a’ starting point for. forecasting future. ~ — 
demand by providing a-profile of. user characteristics: Knowledge = 7. | 
“of current responsiveness, together with.current. user evaluations, oe 
oo provides} the-basis for’ strengthening or relaxing such requirements ~ °° 
fin. dnt the 'f ture.. Both buyer. and seller should benefit from having 
pee . 8 Straightforward. way to ‘determine ifeontract terms are being 
eo met PL ally,. these measurement and analysis. techniques can be 99.0. 
Yo applied to installed systems to determiine’.the level of service- po tee 
© a £  Bedng. pr tho pM tentity where! improvements are needed, and'to 
Us". .y ewaluatel the effects of changes, made.to the system; * 44 tha 


sJ-) Application. of the-methodolo 


oh gy to propoged system could be”, 
oe ptr of lany -benohmarxi g evaluation to determine if responsive- a ee 
“Tness i's bs elatmed- in: Une bid? After .selection.and procur’ement, . ae cn 
. * "2. the methodology could continually bé.used' to ensure compliance 
7 * #cwith contractual. provisions. This woyld-. require.that perfor- . 
7"; mance requireménts b@@specified in the terms of the contract -- 4 >. 
- a-practice that is not.too common ‘today. The: stati tisal ‘adie. | 


1. developed:-as’a part of this ‘dissertation could, be uged.to: © 
by (demonstrate non- comp] gce with contyadturat, sérvice «- : | 
a tions, “0 yf. oe he Pe ae ee, ae a 
; en ae . - aoe aks 4 < 7 ae cae eae 3 ; st = _ 6 WN, 2 _ . ar con” Pe ; , a 
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ly WMI fact, similar techniques (based on the same measurement"... 
uo", device used "in this study) have.alread ‘been. applied in the aan 
_ Procurement’ process (Abrams and. Hayden, 1978), but without: thg - e 

_ full statistical rigor of the methodology Javelopéd here, 7". 
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“ons vderabie data’ ‘end terizing terminal users. has been |. 


“collected, Knowledge of the distributions for such parameters as »- . 
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“the. users... - 


hte scientific Limesharing users employing: m 
data has Been ompared with widely circula 
_ Jackson, Stubbs and Fuchs. (Jackson. and St bbs, 1969; | 


_ of these sé&ts of data should be ‘taken as’ 


“-applicati could to employ a i ethodology. to collect’: 


- Gonditions that led: to, many of the recorded conversation being 


as 2. 3. ining Interactive systems i ee . ae 


oe System tuning eaters to the hates of nein small chaneese 
in. -systlem ‘hardware -or software.-in-the effort to optimize. system- 
" performance (suck as by eliminating bottlenecks). For tuning: 
“interactive. systems, installation managers need a way. . to. assess ~ 
the impact: of systém modifications, on a Bg e to users. These 


effects of system. changes on system pérformance. By taking 

‘ meapurements before and after: a syste modification which. ee ae 
expected. to. impfove performance,’ the. level of improvement - “that ty sh 
re lized - can. be’ ‘measur ed. ‘Thus, an objective. awéasure af service.|\. - 
- improvement can be provided to ij Stallation management for their: | 


use. in’ determining. whether, the. gryeources expended were justified 
ae the results. realized. :, 


‘shared | under the prinaiple that. 


‘typing. rate and length of message are essential to the design’ ans : 
modern computer communications systeps.. Most gurrent - ot 
communications networks and timeshariing: Systems are: designed: for 
sers arg not ‘active all the. 
who: ean point eabecuety 
e degradation of. 7 
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time. - Th numbe'r of -interactive user 
be served "in this fashion (without, extr 
service) iS.a’ peat ierieal, function. of ‘the 
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ern. terminals. - ‘This — 


d data coxlected by. 


Descriptive data. hae been’ “ob aineqe for. 


Jackson,.1970) and differences have been: noted. ‘Howe 


interactive. applications. Qesigners of gystems. for. Bye of Si 


data’ descriptive ‘of. users. engaged in: tasks: more AyPicas., of their 
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Other. types of’ user- Jorientel factors ‘could eisai be oe 
investigated with: these techniques. ‘For example,’ the error | 


discarded. could: be: analyzed ‘in detail. .With greater contrg@l 
and/or: knowledge of the users’ heing measured, @ variety of human 
factors experiments | can be’ conducted, “including . investigation. of. 


-the effects of. user’ experience. (Walther and ‘O'Neil, 1974)-,--uger: af ed 
-. learning rate (Jutdila and Bar am ; Dae different berm nay: a 
ic types (Walther. and o'Neal, irs 
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techniques can. .be applied: ‘to evaluating in a quantitative: wayef 


“the. ‘evaluation vecin.ayes, can 1 be copied: fon: either. hneraware Ce 
software changes, and evén to déhermine if software” a ee oe 
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Le subsystems (exg., the various language compilers) within a.single oo 
seeyitl cee ot ences in levels of service-can be detected. The ~ 
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characteristigs ofthe different, subsystems can be 


ity expeated to have-an effect on. their value to Users, and ‘may help. - 


— y Pacitlities*by users can provide. guidance as. 
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“to explain “user preferences. -This type of Tnformation’ should 90 = 


- also be of. value to installation management in. all@cating . 


".. maintgnange and “improvement. . 
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results of this study: and. the 


results-of studies’ With terminals: 
,at far greater transmission -speeds (L.°H, Mi ler, 
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1976, 1977), “4b 


' .is'clear that the relatively narrow range of termitial speeds... 


Studies is one limitation of the study, It’ would,be.desirable to ~~ 


perform.a study: that’ applied a yconsistent methodology tg a wide .. 


. tange of terminal ‘speeds, so’ that conclusive results ¢ ld be. 
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_ Obtained. - 7 ra se tN 
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measurement 
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described here offers no direct explariations as té the level. of - 
service provided bythe computer. system being tested. All the 
measurements are: striatly external to the system under 7 
investigati n.” “While these measurements ecould be correlated with - 
k taken ‘internally (e.g., from software monitors) in «.*” 
‘order to gain some additional’ ingight as to the functioning of, 


(the host, ‘no suchicorrelations are propgsedhere.” -_ 
ow See Se ae a ee Ee =. | 
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chema. such as experfence with the system or ime of: day of: - 


% usage. . Rather;- the experfmental ‘design called for-large.gamples | a 
al usdge. . ” 


Of typic _-In' this Way, the--peculiarities of ‘any 
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‘ 


results, +. 


.. Reliance on response. time as the primary measure of. Service - 
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